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Design goals and principles Go ogle 



goals 

■ increase availability 

■ reduce hardware cost 

■ increase flexibility 

■ transparency 

principles 

■ not dependent on specific hardware (e.g. SAN) 

■ support different host systems 

scales linearly with the number of systems 

■ small, iterative development 
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Ganeti overview (1/3) Go ogle 

Ganeti is a software to manage clusters of virtual servers 

• Based on Xen (but not strictly dependent on it) 

• n-node high-availability cluster (future) 

• makes it simple to manage 10s of nodes and 100s of instances 

• software used 

■ language: Python 

■ virtualization: Xen 

- disk management: LVM / DRBD / MD 

■ RPC: Twisted, ssh 
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Ganeti overview (2/3) Google 



Terminology: 

• Cluster 

• Node 

• Master Node 

• Instance 

• Pool 

• Meta-Cluster 
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Ganeti overview (3/3) 
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Ganeti administration (1/4) Go ogle 

The commands are run on the master node 

• gnt-node: add / remove / list cluster nodes 

• gnt-instance: 

■ add / remove instance 

■ failover instance, change secondary 

■ stop / start instance, change parameters 

• gnt-os: instance OS definitions 

• gnt-cluster: cluster commands 

• gnt-backup: instance export and import 

All commands have man pages and support interactive help. 
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Ganeti administration (2/4) Go ogle 

Cluster Setup: 

nodeO# gnt-cluster init mycluster 

nodeO# gnt-node add nodel 

nodeO# gnt-node add node2 

node0# gnt-node add node3 

node0# gnt-cluster command \ 

> apt-get install ganeti-instance-etch 
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Ganeti administration (3/4) Go ogle 

Creation of an instance: 

nodeO# gnt-instance add \ 

> -n node2:nodel \ 

> -t drbd8 \ 

> instanceO 

Migration after a node crash: 

nodeO# gnt-instance failover - -ignore-consistency instanceO 
nodeO# gnt-instance replace-disks -s \ 

> - -new-secondary=node3 instanceO 
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Ganeti administration (4/4) 



Google 



Cluster status: 

# gnt-instance list 

Instance OS Primarynode Autostart Status Memory 

instancel.example.com etch nodel.example.com yes running 128 

instance2.example.com etch node3.example.com yes running 512 

instance3.example.com etch node3.example.com yes running 1024 

instance4.example.com etch node2.example.com yes running 128 

instance5.example.com etch node4.example.com yes running 512 



# gnt-node list 

Node 

nodel.example.com 

node2.example.com 

node3.example.com 

node4 . example . com 



DTotal 
858240 
572160 
858240 
356032 



DFree 
442752 
567296 
858240 
356032 



MTotal 
4095 
4095 
4095 
4095 



MNode 
511 
511 
511 
511 



MFree 
3456 
3456 
2048 
3072 



Pinst 
1 
1 
2 
1 



Sinst 
2 
2 
1 
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Instance failover (1/4) 



Google 
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Instance failover (2/4) 



Google 
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Instance failover (3/4) 



Google 
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Instance failover (4/4) 



Google 
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Ganeti disk details 



Google 
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Ganeti remote_raid1 disk recovery 



Google 
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Ganeti remote_raid1 disk recovery 



Google 



remote_raid1 failover 
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Ganeti remote_raid1 disk recovery 



Google 
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Ganeti remote_raid1 disk recovery 



Google 
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Optional advanced features 



Google 



Separate replication network 
Multiple bridges/VLAN support 
Tagging (new) 
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ethl 
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Ganeti usage in Google 



Google 
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20-node Ganeti cluster 

64-bit node OS 

80 virtual instances 

used for internal systems 

not used for google.com 

best for non-resource intensive 
systems 
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Ganeti code Google 



developed at Google 

license: GPLv2 

code location: http://code.google.eom/p/ganeti/ 

August 2007 

■ open source and release 1 .2b1 

November 2007 

■ release 1 .2b2 

December 2007 

■ release 1 .2 

February 2008 

■ release 1.2.1 

Later 

■ release 1 .3 
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1.2 Road m a p Google 



Release 1.2b2: 

■ new cluster configuration format 

■ drbc!8 disk template 

simplify common tasks (node evacuation, reboot, tags) 

■ ganeti-watcher now reactivates drbd pairs 
* easier packaging experience 

tags 

Release 1.2: 

■ no more new features 

■ code cleanup and bugfixes 

Future point releases: 

■ only features that do not affect the core code 

■ investigate experimental support for KVM and Xen-HVM 



Copyright by Google Inc 24 



1 .3 Draft Roadmap Google 



External API 

Transparent failover 

Granular locking 

Job Queuing 

Support for more diverse instances 

Stable support for different virtualization technologies 
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The Future 



Google 



automatic instance failover 

automatic node allocation 

master node election 

manager GUI / meta-cluster manager 



location A 



location B 



20-node 
Ganeti cluster 



20-node 
Ganeti cluster 



30-node 
Ganeti cluster 




manager and 
allocator 
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